<html>
<head><meta charset="utf-8"><title>Eager expansion of built-in macros · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html">Eager expansion of built-in macros</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="180420206"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180420206" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180420206">(Nov 11 2019 at 12:50)</a>:</h4>
<p>Certain built-in macros expand arguments <em>before</em> expanding the macro itself, which is very different from the way macros usually work. The most common use for this feature is probably</p>
<div class="codehilite"><pre><span></span> include!(concat!(env!(&quot;OUT_DIR&quot;), &quot;/bindings.rs&quot;));
</pre></div>


<p>pattern, where both <code>concat!</code> and <code>include!</code> "evaluate" arguments before doing their thing. </p>
<p>Is there any documentation/interesting PR about this behavior? How it interacts with name resolution?</p>
<p>cc <span class="user-mention" data-user-id="123856">@Vadim Petrochenkov</span>, <span class="user-mention" data-user-id="216201">@Edwin Cheng</span></p>



<a name="180421091"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180421091" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180421091">(Nov 11 2019 at 13:01)</a>:</h4>
<p>Maybe off-topic : this is a RFC for general eager expansion  <a href="https://github.com/rust-lang/rfcs/pull/2320" target="_blank" title="https://github.com/rust-lang/rfcs/pull/2320">https://github.com/rust-lang/rfcs/pull/2320</a></p>



<a name="180447690"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180447690" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180447690">(Nov 11 2019 at 18:32)</a>:</h4>
<p>The interesting PR happened in 2013 to support a specific use case - <a href="https://github.com/rust-lang/rust/pull/9740" target="_blank" title="https://github.com/rust-lang/rust/pull/9740">https://github.com/rust-lang/rust/pull/9740</a>.</p>



<a name="180447730"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180447730" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180447730">(Nov 11 2019 at 18:33)</a>:</h4>
<p>jbclements was a hygiene expert back then, and here's his reaction to that PR - <a href="https://github.com/rust-lang/rust/pull/9740#issuecomment-28265862" target="_blank" title="https://github.com/rust-lang/rust/pull/9740#issuecomment-28265862">https://github.com/rust-lang/rust/pull/9740#issuecomment-28265862</a>.</p>



<a name="180447824"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180447824" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180447824">(Nov 11 2019 at 18:34)</a>:</h4>
<p>Since then and until 1.0 a bunch of code depended on this in practice and this is how eager expansion got into stable Rust.</p>



<a name="180448000"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180448000" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180448000">(Nov 11 2019 at 18:37)</a>:</h4>
<blockquote>
<p>How it interacts with name resolution?</p>
</blockquote>
<p>Eagerly expanded macros (and also macros eagerly expanded by eagerly expanded macros, which actually happens in practice too!) are resolved at the location of the "root" macro that performs the eager expansion on its arguments.</p>



<a name="180448145"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180448145" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180448145">(Nov 11 2019 at 18:39)</a>:</h4>
<p>If some name cannot be resolved at the eager expansion time it's considered unresolved, even if becomes available later (e.g. from a glob import or other macro).</p>



<a name="180448231"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180448231" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180448231">(Nov 11 2019 at 18:40)</a>:</h4>
<p>Eagerly expanded macros don't add anything to the module structure of the crate and don't build any speculative module structures, i.e. they are expanded in a "flat" way even if tokens in them look like modules.</p>



<a name="180448347"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180448347" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180448347">(Nov 11 2019 at 18:42)</a>:</h4>
<p>In other words, it kinda works for simple cases for which it was originally intended, and we need to live with it because it's available on stable and widely relied upon.</p>



<a name="180832057"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180832057" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180832057">(Nov 15 2019 at 13:37)</a>:</h4>
<blockquote>
<p>Eagerly expanded macros don't add anything to the module structure</p>
</blockquote>
<p>To clarify, this means that it's impossible to define items with eager macros? I.e, there's no way to sneak in an <code>impl</code> via <code>concat_idents!</code>?</p>



<a name="180833749"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180833749" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180833749">(Nov 15 2019 at 13:56)</a>:</h4>
<p>In general, it seems like proper eager expansion throws a wrench into the incremental name resolution. At the moment, we incrementalise nameres by making each macro expansion a query, whose arguments are a macro def id and a token tree. By resolving the macro, we condense all resolver context into a single integer. For eager expansion, however, we would need the full set of visible  names as a query argument. </p>
<p>Cc <span class="user-mention" data-user-id="116009">@nikomatsakis</span> just in case, as they suggested the way to handle non-eager macros incrementally.</p>



<a name="180835256"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180835256" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180835256">(Nov 15 2019 at 14:12)</a>:</h4>
<p>Or perhaps it's not that bad, if we just use a function to expand an eager-macro, and then just intern the resulting token tree? Ie, we will be interning really large and complex values. Ie, using interning not like traditional space-saving interning, but just like a key-value store</p>



<a name="180845529"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180845529" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180845529">(Nov 15 2019 at 15:59)</a>:</h4>
<p>oh, cc <span class="user-mention" data-user-id="123856">@Vadim Petrochenkov</span> :)</p>



<a name="180915175"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180915175" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180915175">(Nov 16 2019 at 14:58)</a>:</h4>
<blockquote>
<p>To clarify, this means that it's impossible to define items with eager macros? I.e, there's no way to sneak in an impl via concat_idents!?</p>
</blockquote>
<p>Correct. This also means it's not possible to create temporary items (e.g. additional macros, or modules, or imports) being used only during the expansion.<br>
The eager expansion RFC has a bunch of examples like this - <a href="https://github.com/pierzchalski/rfcs/blob/macro-expansion-api/text/0000-eager-macro-expansion.md#appendix-a-corner-cases" target="_blank" title="https://github.com/pierzchalski/rfcs/blob/macro-expansion-api/text/0000-eager-macro-expansion.md#appendix-a-corner-cases">https://github.com/pierzchalski/rfcs/blob/macro-expansion-api/text/0000-eager-macro-expansion.md#appendix-a-corner-cases</a>.</p>



<a name="180915246"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180915246" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180915246">(Nov 16 2019 at 14:59)</a>:</h4>
<blockquote>
<p>In general, it seems like proper eager expansion throws a wrench into the incremental name resolution.</p>
</blockquote>
<p>Non-proper eager expansion seems to break incremental (as you are describing it) as well.<br>
Since the macro needs to resolve names in its arguments, the current resolver state becomes the macro's input.</p>



<a name="180915834"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/180915834" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#180915834">(Nov 16 2019 at 15:16)</a>:</h4>
<blockquote>
<p>If some name cannot be resolved at the eager expansion time it's considered unresolved, even if becomes available later (e.g. from a glob import or other macro).</p>
</blockquote>
<p>I think this part is fixable without too much complications.<br>
The macro's output can be changed from <code>TokenStream</code> to <code>Result&lt;TokenStream, Indeterminate&gt;</code> (internally), and if the macro returns <code>Indeterminate</code> it just returns to the queue to be tried again later.</p>
<blockquote>
<p>Eagerly expanded macros don't add anything to the module structure of the crate and don't build any speculative module structures, i.e. they are expanded in a "flat" way even if tokens in them look like modules.</p>
</blockquote>
<p>And this is something we may want to leave as is - explicitly specify eager expansion as being "flat" / "token-only" and tell users to deal with it.<br>
Otherwise implementing speculative module structures is a big serious work, and work like this very rarely gets done, see <a href="https://github.com/rust-lang/rust/issues/43081" target="_blank" title="https://github.com/rust-lang/rust/issues/43081">#43081</a> (unless perhaps some payed person/people are focusing on it full-time).</p>



<a name="181143121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/181143121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#181143121">(Nov 19 2019 at 19:00)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> I'm not quite clear on whether this got "resolved" here</p>



<a name="181143158"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/181143158" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#181143158">(Nov 19 2019 at 19:00)</a>:</h4>
<p>Let me resummarize to be sure I understand</p>



<a name="181143204"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/181143204" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#181143204">(Nov 19 2019 at 19:00)</a>:</h4>
<p>the core problem here is that expanding <em>certain macros</em> (e.g., <code>include!</code>) "eagerly evaluate" their arguments</p>



<a name="181143224"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/181143224" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#181143224">(Nov 19 2019 at 19:01)</a>:</h4>
<p>right now, in rust-analyzer, expanding macros in general is a query</p>



<a name="181143247"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/181143247" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#181143247">(Nov 19 2019 at 19:01)</a>:</h4>
<p>and that query doesn't have access to enough state to do that eager evaluation</p>



<a name="181143253"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/181143253" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#181143253">(Nov 19 2019 at 19:01)</a>:</h4>
<p>correct?</p>



<a name="181145977"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Eager%20expansion%20of%20built-in%20macros/near/181145977" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Eager.20expansion.20of.20built-in.20macros.html#181145977">(Nov 19 2019 at 19:28)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> , correct!</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>